Every day, important scientific findings are rejected at large. From man-made climate change to the safety and efficacy of Covid-19 vaccinations, science skepticism has run rampant among lay consumers in modern society (Hornsey & Fielding, 2017). To increase public faith in science, some have proposed the use of crowd science (Silberzahn et al., 2018; Uhlmann et al., 2019).
\(~~~~~\) We explore the effects of scientific findings emerging from a crowd of researchers (vs. a typical research collaboration) on lay perceptions of scientific findings. In line with social norm theory (Miller & Prentice, 2016), we expect that observing consensus among a crowd (the consistent crowd condition) will – compared to the conclusion of a single scientist (the single estimate condition) – increase conformity in opinion. Drawing from work on intuitive statistics (Gigerenzer & Murray, 2015), we also expect laypeople to intuitively accord to the logic of the wisdom of crowds: the ability of an aggregate of multiple estimates (rather than a single estimate) to reduce noise stemming from individual bias or error (Schweinsberg et al., 2021).
\(~~~~~\) In contrast, when crowd estimates show low consensus and high variance (the inconsistent crowd condition), we predict that observers will be less swayed and more likely to attribute the findings to bias and error. In addition, due to the difficulty of lay reasoning about variation (Ben-Zvi & Garfield, 1999), we predict an aversion to variability: i.e., we expect that observing variable estimates will decrease lay confidence in the precise average parameter estimate in both crowd conditions.
Table 1: Predicted differences with the single estimate condition
| Measure | Consistent crowd | Inconsistent crowd |
|---|---|---|
| 1. Posterior beliefs in the phenomenon | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 2. Credibility of the results | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 3. Confidence in the precise estimate | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 4. Scientific bias | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 5. Scientific error | \(~~~~~~~~~~\) | \(~~~~~~~~~~\) |
| 6. Scientific discretion | \(~~~\) No prediction | \(~~~\) No prediction |
Note. We regress each outcome on prior beliefs and condition (with the single estimate condition as the reference category). When laypeople observe multiple consistent (inconsistent) estimates from a crowd, we expect – compared to a single estimate and controlling for prior beliefs – higher (lower) posterior beliefs and credibility of the results, lower confidence in the precise average parameter estimate, and lower (higher) ratings of bias and error.
Open Science: Preregistration, survey, data, and code available at
\(~~\) github.com/shilaan/many-analysts
\(~~\) osf.io/vedb4
We ran an experiment (N = 1,498; UK/US Prolific) with three conditions
\(~\) Single estimate
A single parameter estimate (5%)
\(~\) Consistent crowd
Multiple crowd estimates: low variance, high consensus (M = 5%)
\(~\) Inconsistent crowd
Multiple crowd estimates: high variance, low consensus (M = 5%)
Experimental Design
Figure 1: Estimates of differences with the single estimate condition
In line with our hypotheses, lay consumers of inconsistent crowd estimates (vs. a single estimate)…
\(~~\) Have lower posterior beliefs about the reported phenomenon
\(~~\) Find the results less credible
\(~~\) Have less confidence in the average estimate of 5%
\(~~\) Are more likely to attribute the average estimate (5%) to bias
\(~~\) Are more likely to attribute the average estimate (5%) to error
Contrary to our hypotheses, lay consumers of consistent crowd estimates (vs. a single estimate)…
\(~~\) Have lower posterior beliefs about the reported phenomenon
\(~~\) Are more likely to attribute the average estimate (5%) to error
We found no significant effects for lay consumers of consistent crowd estimates (vs. a single estimate) on…
\(~~\) Credibility of the results
\(~~\) Confidence in the average estimate
\(~~\) Ratings of bias
Exploratory results
For the additional exploratory measure, lay consumers of consistent and inconsistent crowd estimates…
\(~~\) Perceive greater discretion (i.e., idiosyncratic choices)
Figure 2: Distribution of prior and posterior beliefs by condition
In terms of belief updating, Figure 2 shows a positive difference within the consistent crowd condition \((\)pre vs. post \(M_d\) = 4.75 [2.55,6.95]\()\), but less so than for the single estimate condition \((M_d\) = 11.66 [9.66,13.66]\()\). As expected, we find negative belief updating in the inconsistent crowd condition \((M_d\) = -11.45 [-13.75,-9.16]\()\).
\(~~\) Compared to providing a single estimate, we find no evidence that
\(~~~~~~~\) crowd estimates improve lay perceptions of scientific findings
Future directions
\(~~\) Does variability aversion explain the findings?
\(~~~\) Perceptions of scientists
\(~~\) Science communication and communicating uncertainty